Image processing apparatus, image processing method, computer program for processing images, and recording medium

ABSTRACT

An image processing device that, even when a degradation model differs from a real degradation process of an obtained low-resolution image, generates a high-resolution image while restricting noises from occurring due to differences between the degradation model and the real degradation process. The image processing device includes: an enlargement interpolation unit generating a first high-resolution image that is higher in resolution than the low-resolution image; a super resolution processing unit generating, from the low-resolution image, a second high-resolution image; a feature generating unit generating features by using the low-resolution image; a difference calculating unit calculating difference values between values of pixels; an adjustment unit calculating corrected difference values by correcting the difference values by using the features; and a combination unit generating a third high-resolution image by adding corrected difference values to values of corresponding pixels in the first high-resolution image.

TECHNICAL FIELD

The present invention relates to an image processing technology forgenerating a high-resolution image from a low-resolution image.

BACKGROUND ART

As a method for generating a high-resolution image from a low-resolutionimage, the super resolution technology has been proposed. For example,Non-Patent Literature 1, discloses a representative method of the superresolution technology. According to Non-Patent Literature 1, Equation 1shown below is defined to represent relation between a plurality oflow-resolution images and a high-resolution image, and thehigh-resolution image is generated by solving the Equation 1. To solveEquation 1, iterative processing using the gradient method or the likeis performed, and the calculation is repeatedly performed until theerror becomes constant.

$\begin{matrix}{I = {{\frac{1}{\sigma^{2}}{\sum\limits_{i = 0}^{N_{l - 1}}\left\lbrack {{{b\left( {x_{i},y_{i}} \right)}^{T} \cdot h} - f_{i}} \right\rbrack^{2}}} + {a{{C^{T} \cdot h}}^{2}}}} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

In the above equation, h is a vector representation of a high-resolutionimage, σ is a noise deviation of an observed value, b(xi, yi) is avector representation of a PSF (Point Spread Function) kernelcorresponding to a position (xi, yi), C is a matrix representing priorinformation of the high-resolution image, and α is a constraintparameter representing the strength of the constraint. Also, N_(l) isthe total number of pixels constituting a plurality of low-resolutionimages.

In conventional super resolution technologies, based on an assumptionthat a low-resolution image is a result of a high-resolution imagehaving been degraded for some reason, the relation between a targethigh-resolution image and a low-resolution image is described in anequation, and the target high-resolution image is obtained by solvingthe equation. In obtaining an excellent high-resolution image, theequation as a degradation model plays a vital role, and how to create adegradation model of a low-resolution image based on an assumeddegradation process is a key to the super resolution technology.

When the degradation model differs from the real degradation process ofthe low-resolution image, a noise is included in the obtainedhigh-resolution image due to the difference between the degradationmodel and the real degradation process. Non-Patent Literature 2discloses a super resolution technology that enhances the robustness byadding a member representing a system noise into the degradation model,other than the members representing noises occurring in the process ofinputting the image such as the PSF and the like. According to thetechnology of Non-Patent Literature 2, a degradation model such asEquation 2 is defined, and a high-resolution image is generated bysolving this equation.

Y _(k) =D _(k) H _(k) ^(cam) F _(k) H _(k) ^(atm) X+V _(k)

k=1, . . . ,N  (Equation 2)

In Equation 2, X is an input image, Hatm is an atmospheric blur, Fk is adegradation caused by A/D conversion, Hcam is a camera blur, Dk is adegradation caused by downsampling, and Vk is a system noise. InEquation 2, the degradation including the system noise is modeled. Thusthe equation provides robustness with regard to noises other than thenoises of the regular image degradation.

CITATION LIST Non-Patent Literature Non-Patent Literature 1

-   Masayuki Tanaka and Masatoshi Okutomi, Sai-kousei-gata    chou-kaizou-shori no kousoku-ka arugorizumu to sono seido-hyouka (A    fast algorithm for reconstruction-based superresolution and    evaluation of its accuracy), IEICE Transactions D, Vol. J88-D2, No.    11, pp. 2200-2209, 2005

Non-Patent Literature 2

-   Sina Farsiu et al., Fast and robust multiframe super resolution,    IEEE Transactions on Image Processing, vol. 13, no. 10, October 2004

SUMMARY OF INVENTION Technical Problem

However, in the case where, for example, an attempt is made to apply thesuper resolution to images that are provided by TV broadcasting, inpractice it is impossible to assume the degradation model for each of avast number of broadcast programs. When the technology disclosed inNon-Patent Literature 1 is used to perform the super resolution processwith a degradation model prepared in advance, the high-resolution imageobtained thereby includes a noise, producing an undesirable result.Also, applying the technology disclosed in Non-Patent Literature 2 has aproblem that although robustness is provided to a certain extent, sincethe iterative processing is embedded in Equation 2 to secure therobustness, a large amount of calculation is required.

It is therefore an object of the present invention to provide an imageprocessing device, an image processing method, a computer program forimage processing, and a recording medium that, even when a degradationmodel differs from a real degradation process of an obtainedlow-resolution image, generate a high-resolution image while restrictingnoises from occurring due to differences between the degradation modeland the real degradation process.

Solution to Problem

The above object is fulfilled by an image processing device forgenerating a high-resolution image from a low-resolution image,comprising: an obtaining unit configured to obtain the low-resolutionimage; an enlargement interpolation unit configured to generate a firsthigh-resolution image that is higher in resolution than thelow-resolution image, by performing an enlargement interpolation on theobtained low-resolution image; a super resolution processing unitconfigured to generate, from the low-resolution image, a secondhigh-resolution image that is equal in resolution to the firsthigh-resolution image, by tracing back a process through which an imageis assumed to be degraded in resolution, the process being representedby a degradation model; a feature generating unit configured to generatea feature for each of a plurality of pixels constituting the firsthigh-resolution image by using the low-resolution image; a differencecalculating unit configured to calculate, for each of the plurality ofpixels constituting the first high-resolution image, a difference valuebetween a value of that pixel and a value of a corresponding pixel inthe second high-resolution image; an adjustment unit configured tocalculate a corrected difference value by correcting the calculateddifference value by using the generated feature; and a combination unitconfigured to generate a third high-resolution image by adding correcteddifference values calculated by the adjustment unit to values ofcorresponding pixels in the first high-resolution image.

Advantageous Effects of Invention

As described above, the image processing device of the present inventioncauses the enlargement interpolation unit and the super resolutionprocessing unit to perform different enlargement processes on thelow-resolution image, generate features corresponding to the pixelsconstituting the first high-resolution image from the low-resolutionimage, and corrects the difference values representing differencesbetween values of pixels constituting the first high-resolution imageand the second high-resolution image, by using the generated features.The difference values representing differences between the firsthigh-resolution image and the second high-resolution image include noisecomponents that have occurred in the super resolution process.Accordingly, by correcting the difference values by using the featuresthat have been generated based on the low-resolution image, and addingcorrected difference values, which are obtained by the correction, tothe first high-resolution image, it is possible to restrict noises thatcan be easily perceived.

In this way, the present invention produces an advantageous effect that,when the degradation model used in the super resolution process differsfrom the real degradation process of the obtained low-resolution image,it is possible to restrict noises from occurring and obtain a visuallyexcellent high-resolution image.

The image processing device of the present invention can be usedeffectively in a device for generating a high-resolution image from alow-resolution image, such as a display device that generates ahigh-resolution image from a low-resolution image in compliance with theresolution of the display device, an image output device such as aprinter for outputting a result of enlarging a low-resolution image, anda video shooting device that generates a still image by enlarging one ofa plurality of images constituting a video.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating the structure of the imageprocessing device 10 as one embodiment of the present invention.

FIG. 2 is a block diagram illustrating the structure of the imageprocessing unit 100.

FIG. 3 illustrates one example of pixels to be arranged in the firsthigh-resolution image, the pixels including one central pixel 401 and 36peripheral pixels 411, 412, 413, . . . , 414, . . . , 415.

FIG. 4 illustrates four areas 502, 503, 504 and 505 centering on thepixel 501.

FIG. 5 illustrates the positional relationship among the maximum valuesMAX1, MAX2, MAX3, MAX4, and the attention pixel 501.

FIG. 6 illustrates the positional relationship among the minimum valuesMIN1, MIN2, MIN3, MIN4, and the attention pixel 501.

FIG. 7 illustrates a case where, the change along the x axis of thepixel values of a plurality of pixels arranged in the x-axis directionforms a concave-down shape in the area S.

FIG. 8 illustrates a case where, the change along the x axis of thepixel values of the pixels arranged in the x-axis direction forms aconcave-up shape in the area S.

FIGS. 9A and 9B illustrate the relationship between the gradient and thegain β in the correspondence table included in the adjustment unit 107:FIG. 9A illustrates a first specific example of the relationship betweenthe gradient and the gain β; and FIG. 9B illustrates a second specificexample of the relationship between the gradient and the gain β.

FIG. 10 illustrates one example of the change 1101 in pixel value overpixels arranged in the x axis direction in the second high-resolutionimage, the change 1102 in the maximum value of the permissibleamplitude, and the change 1103 in the minimum value of the permissibleamplitude.

FIG. 11 illustrates one example of the change 1101 in pixel value overpixels arranged in the x axis direction in the second high-resolutionimage, the change 1102 in the maximum value of the permissibleamplitude, the change 1103 in the minimum value of the permissibleamplitude, and the change 1201 in pixel value over pixels arranged inthe x axis direction in the third high-resolution image.

FIG. 12 is a flowchart illustrating the operation of the imageprocessing device 10, centering on the operation of the image processingunit 100.

FIG. 13 is a flowchart illustrating the operation of the enlargementinterpolation unit 105.

DESCRIPTION OF EMBODIMENTS

According to one aspect of the present invention, there is provided animage processing device for generating a high-resolution image from alow-resolution image, comprising: an obtaining unit configured to obtainthe low-resolution image; an enlargement interpolation unit configuredto generate a first high-resolution image that is higher in resolutionthan the low-resolution image, by performing an enlargementinterpolation on the obtained low-resolution image; a super resolutionprocessing unit configured to generate, from the low-resolution image, asecond high-resolution image that is equal in resolution to the firsthigh-resolution image, by tracing back a process through which an imageis assumed to be degraded in resolution, the process being representedby a degradation model; a feature generating unit configured to generatea feature for each of a plurality of pixels constituting the firsthigh-resolution image by using the low-resolution image; a differencecalculating unit configured to calculate, for each of the plurality ofpixels constituting the first high-resolution image, a difference valuebetween a value of that pixel and a value of a corresponding pixel inthe second high-resolution image; an adjustment unit configured tocalculate a corrected difference value by correcting the calculateddifference value by using the generated feature; and a combination unitconfigured to generate a third high-resolution image by adding correcteddifference values calculated by the adjustment unit to values ofcorresponding pixels in the first high-resolution image.

In the above-described image processing device, the feature generatingunit may generate, as the feature, gradient information indicating agradient of a value of a pixel in an image area in the low-resolutionimage corresponding to peripheral pixels of each of the plurality ofpixels constituting the first high-resolution image, and the adjustmentunit may correct the calculated difference value by using the generatedgradient information.

In the above-described image processing device, the adjustment unit maycalculate a gain that increases or decreases depending on the gradientinformation, and calculate the corrected difference value by multiplyingthe calculated gain by the difference value.

In the above-described image processing device, the feature analyzingunit may generate, as the feature, a permissible range from an imagearea in the low-resolution image corresponding to peripheral pixels ofeach of the plurality of pixels constituting the first high-resolutionimage, the permissible range being a range of values that can be takenby that pixel, and the adjustment unit may correct the calculateddifference value for each pixel in the second high-resolution image thathas a value exceeding the permissible range.

According to another aspect of the present invention, there is providedan image processing method for use in an image processing device forgenerating a high-resolution image from a low-resolution image,comprising: an obtaining step of obtaining the low-resolution image; anenlargement interpolation step of generating a first high-resolutionimage that is higher in resolution than the low-resolution image, byperforming an enlargement interpolation on the obtained low-resolutionimage; a super resolution processing step of generating, from thelow-resolution image, a second high-resolution image that is equal inresolution to the first high-resolution image, by tracing back a processthrough which an image is assumed to be degraded in resolution, theprocess being represented by a degradation model; a feature generatingstep of generating a feature for each of a plurality of pixelsconstituting the first high-resolution image by using the low-resolutionimage; a difference calculating step of calculating, for each of theplurality of pixels constituting the first high-resolution image, adifference value between a value of that pixel and a value of acorresponding pixel in the second high-resolution image; an adjustmentstep of calculating a corrected difference value by correcting thecalculated difference value by using the generated feature; and acombination step of generating a third high-resolution image by addingcorrected difference values calculated by the adjustment unit to valuesof corresponding pixels in the first high-resolution image.

According to a still another aspect of the present invention, there isprovided a computer program for image processing for use in an imageprocessing device for generating a high-resolution image from alow-resolution image, the computer program causing a computer toexecute: an obtaining step of obtaining the low-resolution image; anenlargement interpolation step of generating a first high-resolutionimage that is higher in resolution than the low-resolution image, byperforming an enlargement interpolation on the obtained low-resolutionimage; a super resolution processing step of generating, from thelow-resolution image, a second high-resolution image that is equal inresolution to the first high-resolution image, by tracing back a processthrough which an image is assumed to be degraded in resolution, theprocess being represented by a degradation model; a feature generatingstep of generating a feature for each of a plurality of pixelsconstituting the first high-resolution image by using the low-resolutionimage; a difference calculating step of calculating, for each of theplurality of pixels constituting the first high-resolution image, adifference value between a value of that pixel and a value of acorresponding pixel in the second high-resolution image; an adjustmentstep of calculating a corrected difference value by correcting thecalculated difference value by using the generated feature; and acombination step of generating a third high-resolution image by addingcorrected difference values calculated by the adjustment unit to valuesof corresponding pixels in the first high-resolution image.

According to a further aspect of the present invention, there isprovided a computer-readable recording medium storing a computer programfor image processing for use in an image processing device forgenerating a high-resolution image from a low-resolution image, thecomputer program causing a computer to execute: an obtaining step ofobtaining the low-resolution image; an enlargement interpolation step ofgenerating a first high-resolution image that is higher in resolutionthan the low-resolution image, by performing an enlargementinterpolation on the obtained low-resolution image; a super resolutionprocessing step of generating, from the low-resolution image, a secondhigh-resolution image that is equal in resolution to the firsthigh-resolution image, by tracing back a process through which an imageis assumed to be degraded in resolution, the process being representedby a degradation model; a feature generating step of generating afeature for each of a plurality of pixels constituting the firsthigh-resolution image by using the low-resolution image; a differencecalculating step of calculating, for each of the plurality of pixelsconstituting the first high-resolution image, a difference value betweena value of that pixel and a value of a corresponding pixel in the secondhigh-resolution image; an adjustment step of calculating a correcteddifference value by correcting the calculated difference value by usingthe generated feature; and a combination step of generating a thirdhigh-resolution image by adding corrected difference values calculatedby the adjustment unit to values of corresponding pixels in the firsthigh-resolution image.

Embodiment

The following describes an image processing device 10 as one embodimentof the present invention with reference to the accompanying drawings.

1. Image Processing Device 10

As illustrated in FIG. 1, the image processing device 10 includes aninput unit 201, a tuner 202, an external terminal 203, a memory 205, aprocessor 206, a decoding unit 207, a display control unit 208, a videodriver 209, and a hard disk drive 210. The image processing device 10 isconnected with a display device 211 that uses liquid crystal, plasma,CRT or the like. A memory card 204 is attachable to the image processingdevice 10.

The image processing device 10, which may be a hard disk recorder as oneexample, stores one or more broadcast programs received via the digitalbroadcasting, and plays back one of the stored programs on the displaydevice 211 as instructed by the user. The display device 211 is, as oneexample, a large-scale television display having a high resolution fordisplaying the programs.

The tuner 202 extracts a TV signal from received broadcast waves, andoutputs the extracted TV signal to the input unit 201. The TV signalincludes image data. The external terminal 203 receives a video signalfrom another video playback device (not illustrated) such as a DVDplayback device, and outputs the received video signal to the input unit201. The video signal includes image data. The memory card 204 is, asone example, an SD card and stores videos and/or still images.

The input unit 201 receives image data (hereinafter referred to as“low-resolution image”) from the tuner 202, external terminal 203, ormemory card 204. Here, the low-resolution image is an image whoseresolution is lower than the resolution of the display device 211. Thememory 205 is used as a primary storage of the image processing device10, and is composed of a DRAM or the like. The processor 206 controlsthe image processing device 10 as a whole. The hard disk drive 210 is asecondary storage for storing images and the like, and storeslow-resolution images input from the input unit 201.

When a low-resolution image received by the input unit 201 is acompressed image, or when a low-resolution image stored in the hard diskdrive 210 is a compressed image, the decoding unit 207 decodes thecompressed image. Decoded images are sent to the display control unit208 directly or via the memory 205. The display control unit 208converts a received image into an image having a resolution that issupported by the display device 211 (hereinafter the image is referredto as a “third high-resolution image”), and outputs the thirdhigh-resolution image to the video driver 209. The video driver 209outputs the third high-resolution image to the display device 211 andcontrols display of the third high-resolution image on the displaydevice 211.

Note that, according to the above description, the image processingdevice 10 includes the tuner 202. However, the present invention is notlimited to this. The image processing device 10 may not include thetuner 202, but may be connected with a digital broadcast receivingdevice that includes a tuner.

The display control unit 208 includes an image processing unit 100illustrated in FIG. 2 for generating the third high-resolution image byconverting a received low-resolution image to have a resolution thatmatches the resolution of the display device 211.

The display control unit 208 extracts, from a decoded low-resolutionimage, a value in XS indicating the number of pixels in the horizontaldirection and a value in YS indicating the number of pixels in thevertical direction of the low-resolution image. The display control unit208 also receives, from the video driver 209, a value outXS indicatingthe number of pixels in the horizontal direction and a value outYSindicating the number of pixels in the vertical direction of the thirdhigh-resolution image to be displayed on the display device 211.

The image processing unit 100 is implemented as hardware. However, notlimited to this, the image processing unit 100 may be implemented assoftware processing of the processor 206. Note that, in either case ofimplementation, the image processing unit 100 can be used, for example,to re-encode a generated third high-resolution image and write it backonto the hard disk drive 210.

2. Image Processing Unit 100

The image processing unit 100, as illustrated in FIG. 2, includes afeature generating unit 103, a super resolution enlargement unit 104, anenlargement interpolation unit 105, a difference calculating unit 106,an adjustment unit 107, and a combination unit 108.

The image processing unit 100 receives a low-resolution image.

The enlargement interpolation unit 105 generates a first high-resolutionimage that is higher in resolution than the received low-resolutionimage, by performing an enlargement interpolation on the receivedlow-resolution image. The super resolution enlargement unit 104generates, from the received low-resolution image, a secondhigh-resolution image that is equal in resolution to the firsthigh-resolution image, by tracing back a process through which an imageis assumed to be degraded in resolution, the process being representedby a degradation model. The feature generating unit 103 generates afeature for each of a plurality of pixels constituting the firsthigh-resolution image, based on the received low-resolution image. Thedifference calculating unit 106 calculates, for each of the plurality ofpixels constituting the first high-resolution image, a difference valuebetween a value of that pixel and a value of a corresponding pixel inthe second high-resolution image. The adjustment unit 107 calculates acorrected difference value by correcting each of the calculateddifference values based on the feature generated as described above. Thecombination unit 108 generates the third high-resolution image by addingthe corrected difference values respectively to the values of the pixelsconstituting the first high-resolution image, and outputs the generatedthird high-resolution image.

The image processing unit 100 receives the value in XS indicating thenumber of pixels in the horizontal direction and the value in YSindicating the number of pixels in the vertical direction of thelow-resolution image, and receives the value outXS indicating the numberof pixels in the horizontal direction and the value outYS indicating thenumber of pixels in the vertical direction of the high-resolution image.

(1) Enlargement Interpolation Unit 105

The enlargement interpolation unit 105 generates the firsthigh-resolution image as follows. Note that the first high-resolutionimage and the third high-resolution image to be displayed on the displaydevice 211 have the same resolution, number of pixels in horizontaldirection and number of pixels in vertical direction.

The enlargement interpolation unit 105 receives the low-resolutionimage. The enlargement interpolation unit 105 also receives the value inXS indicating the number of pixels in the horizontal direction and thevalue in YS indicating the number of pixels in the vertical direction ofthe low-resolution image, and receives the value outXS indicating thenumber of pixels in the horizontal direction and the value outYSindicating the number of pixels in the vertical direction of the thirdhigh-resolution image.

Subsequently, the enlargement interpolation unit 105 calculates aninterpolation interval dx in the horizontal direction and aninterpolation interval dy in the vertical direction, based on thefollowing Equations 3 and 4, using the values in XS, in YS, outXS, andoutYS each indicating the number of pixels.

Interpolation interval dx in the horizontal direction=inDX/outDX  (Equation 3)

Interpolation interval dy in the vertical direction=inDY/outDY  (Equation 4)

The interpolation interval dx in the horizontal direction and theinterpolation interval dy in the vertical direction, calculated asdescribed above, define the positions of the pixels constituting thefirst high-resolution image. That is to say, in the firsthigh-resolution image, each pixel is arranged at a position that is aninteger multiple of the horizontal interpolation interval dx in thehorizontal direction, and is arranged at a position that is an integermultiple of the vertical interpolation interval dy in the verticaldirection.

Note that since the low-resolution image is lower in resolution than thethird high-resolution image, values dx and dy are each smaller than 1.

Subsequently, the enlargement interpolation unit 105 repeatedly performsthe following processes (a) to (c) for each of all the pixels to bearranged in the first high-resolution image.

(a) With regard to the position (referred to as “central position”) ofone pixel (referred to as “central pixel”) to be arranged in the firsthigh-resolution image, the enlargement interpolation unit 105 selects,as one example, 36 pixels (referred to as “peripheral pixels”) arrangedin a matrix of six rows and six columns, from among a plurality ofpixels constituting the low-resolution image such that the centralposition is arranged approximately at the center of the peripheralpixels, and obtains pixel values of the selected peripheral pixels.

FIG. 3 illustrates one example of pixels arranged, the pixels includingone central pixel 401 of the first high-resolution image and 36peripheral pixels 411, 412, 413, . . . , 414, . . . , 415 selected fromthe low-resolution image. As illustrated in FIG. 3, the central pixel401 is placed approximately at the center of the peripheral pixels 411,412, 413, . . . , 414, . . . , 415 arranged as a matrix of six rows andsix columns.

In FIG. 3, the coordinates (x, y) of the central pixel 401 are, forexample, (3.4, 3.5).

(b) Subsequently, respective decimal parts px and py of the x and ycoordinates of the central position are calculated.

As illustrated in FIG. 3, decimal parts px and py are components of thedistance between the central pixel 401 and a peripheral pixel 414 thatis nearest to the central pixel 401, in the x-axis and y-axisdirections, respectively. Note that, in FIG. 3, px=0.4 and py=0.5, asone example.

(c) Subsequently, by using the obtained pixel values of the peripheralpixels and decimal parts px and py, the pixel value of the central pixelat the central position is calculated by the enlargement interpolationmethod, and the calculated pixel value is stored at the central positionof the first high-resolution image.

The enlargement interpolation method for use may be the bilinear method,bicubic convolution method, or Lanczos method that are conventionallyknown techniques. For example, in the bicubic convolution method, theinterpolation calculation is performed based on 4×4 pixels locatedaround a point (x, y).

As one example, a pixel value “out” of the central pixel is calculatedbased on the following Equation 5.

$\begin{matrix}{{out} = \frac{\sum\limits_{l = 1}^{4}\; {\sum\limits_{m = 1}^{4}\left( {{I\left( {l,m} \right)} \times w\; x_{l} \times w\; y_{m}} \right)}}{\sum\limits_{l = 1}^{4}{\sum\limits_{m = 1}^{4}\left( {w\; x_{l} \times w\; y_{m}} \right)}}} & \left( {{Equation}\mspace{14mu} 5} \right)\end{matrix}$

In the above equation, out is an output pixel value, I(l,m) is pixelvalues of 4×4 pixels at position (l,m), wxl is a weight coefficient atposition/in the x direction, and wym is a weight coefficient at positionm in the y direction. Note that each of the weight coefficients isrepresented by the following Equation 6 in the case of the bicubicinterpolation, where d is the distance from the interpolation pixelposition represented by px, py in the x or y direction.

$\begin{matrix}{w = \left\{ \begin{matrix}{1 - {2\; d^{2}} + d^{3}} & {d \leq 1.0} \\{4 - {8\; d} + {5\; d^{2}} - d^{3}} & {1.0 < d \leq 2.0} \\0 & {d > 2.0}\end{matrix} \right.} & \left( {{Equation}\mspace{14mu} 6} \right)\end{matrix}$

Also, w is represented by the following Equation 7 in the case of theLanczos method.

$\begin{matrix}{w = \left\{ \begin{matrix}{\frac{\sin \left( {d\; \pi} \right)}{\pi}\frac{\sin \left( {\frac{x}{2}\pi} \right)}{\pi^{\frac{d}{2}}}} & {d < 2} \\0 & {d \geq 2}\end{matrix} \right.} & \left( {{Equation}\mspace{14mu} 7} \right)\end{matrix}$

(2) Super Resolution Enlargement Unit 104

Let Y denote a low-resolution image that is input, and X an unknown,correct second high-resolution image, and then input image Y can beconsidered to be a result of degradation represented by degradationfunction H. The following Equation 8 represents this relationship.

Y=HX  (Equation 8)

Note that the degradation function H includes a reduction process forconverting the second high-resolution image into a low-resolution image,a PSF (Point Spread Function), etc.

The super resolution enlargement unit 104 performs a process forobtaining X (namely the second high-resolution image) by performing areverse calculation of Equation 8.

Note that noises other than the degradation and constraint condition Vmay be added to Equation 8, as defined by the following Equation 9.

Y=HX+V  (Equation 9)

In the reverse calculation of this equation, in general, a secondhigh-resolution image X is temporarily determined and Y′ is calculatedbased on Equation 8 or 9, and these calculations are repeatedlyperformed until the difference between the result Y′ and the input imageY becomes smaller than a threshold value. As another method, X may becalculated by using a plurality of frames constituting a video that isinput as the low-resolution image. Further, as a simple method, X may beobtained by deconvolution.

In any case, the super resolution enlargement process of the presentembodiment includes estimating degradation of an image by apredetermined method, obtaining the reverse function thereof, andgenerating the second high-resolution image.

In the following, a pixel position that is obtained by replacing a pixelposition in the first and second high-resolution images, which arerespectively generated by the enlargement interpolation unit 105 and thesuper resolution enlargement unit 104, with a pixel position (x, y inFIG. 3) in a low-resolution image, is referred to as a “phase”.

If the first high-resolution image output from the enlargementinterpolation unit 105 and the second high-resolution image output fromthe super resolution enlargement unit 104 differ in phase, correspondingpixel values of the two high-resolution images for a same position areshifted from each other, and the subsequent processes performed with thedifference in phase produce incorrect results. Thus the outputs of theenlargement interpolation unit 105 and the super resolution enlargementunit 104 need to be made to coincide with each other in phase. To makethe outputs coincide with each other in phase, pixel positions(interpolation positions x, y, dx, dy) in the first high-resolutionimage output from the enlargement interpolation unit 105 are made to bethe same as pixel positions of corresponding pixels in the secondhigh-resolution image output from the super resolution enlargement unit104.

(3) Feature Generating Unit 103

It is important to estimate the above-mentioned degradation function Hcorrectly since the quality of the obtained second high-resolution imagechanges depending on how the degradation function H is estimated. Notethat in the broadcasting or the like, it is difficult to estimate thedegradation function H correctly. An unintended noise component or thelike occurs if the function H is estimated inappropriately. In thefollowing process of the present embodiment, such unintended noisecomponents are restricted.

The feature generating unit 103 inputs a low-resolution image,calculates features corresponding to pixels constituting a firsthigh-resolution image, and outputs the calculated feature values.Accordingly, the features, which are output as a result of the featureanalysis by the feature generating unit 103, coincide in phase with thefirst and second high-resolution images output from the enlargementinterpolation unit 105 and the super resolution enlargement unit 104,respectively.

The feature generating unit 103 repeatedly performs the followingprocesses (a) to (b) for each of all the pixels to be arranged in thefirst high-resolution image.

(a) With regard to the position (referred to as “attention position”) ofone pixel (referred to as “attention pixel”) to be arranged in the firsthigh-resolution image, the feature generating unit 103 selects, as oneexample, N×M pixels arranged in a matrix of horizontal N pixels andvertical M pixels (an area composed of these pixels is referred to as an“area S”), from among a plurality of pixels constituting thelow-resolution image such that the attention position is arrangedapproximately at the center of the area S, and obtains pixel values ofthe selected peripheral pixels.

(b) The feature generating unit 103 calculates, as the features, thegradient and the permissible amplitude of the area S centered on theattention pixel.

Note that the area S is composed of N×M pixels centered on theinterpolation phase (x, y) of the low-resolution image, and is composedof 2×2 pixels at the minimum. When the amount of processing and theaccuracy of the features are taken into account, it is desirable that Nand M are each in the range from 3 to 6 approximately.

(Calculation of Gradient of Area S)

The feature generating unit 103 calculates the gradient of the area S inaccordance with //∇u// by regarding the pixel values in the area S ascontinuously changing values in the area S, not as discrete values inthe area S, where u denotes a function of the gradient of the area S.

The //∇u// may be calculated based on the following Equation 10 or 11,for example.

$\begin{matrix}{{{\nabla u}} = \sqrt{\left( \frac{\partial u}{\partial x} \right)^{2} + \left( \frac{\partial u}{\partial y} \right)^{2}}} & \left( {{Equation}\mspace{14mu} 10} \right) \\{{{\nabla u}} = {\frac{\partial u}{\partial x} + \frac{\partial u}{\partial y}}} & \left( {{Equation}\mspace{14mu} 11} \right)\end{matrix}$

Alternatively, for a simplified calculation of //∇u//, the featuregenerating unit 103 may calculate a difference between the maximum valueand the minimum value in the area S.

In any case, regardless of the method for use, the feature generatingunit 103 calculates the gradient in the area S, and outputs thecalculated gradient as one of the features.

(Calculation of Permissible Amplitude)

The permissible amplitude indicates a permissible range for theattention pixel to change. The feature generating unit 103 calculatesthe permissible amplitude from the gradient in the area S or the like.

(i) Simple Method for Calculating Permissible Amplitude

The feature generating unit 103 calculates the maximum value in the areaS as the maximum value of the permissible amplitude, and calculates theminimum value in the area S as the minimum value of the permissibleamplitude.

(ii) Other Methods for Calculating Permissible Amplitude

Here, other methods for calculating the permissible amplitude aredescribed.

(ii-1) First Example

The first example for calculating the permissible amplitude is explainedwith reference to FIGS. 4-6.

In FIG. 4, a pixel 501 is a pixel (attention pixel) to be arranged inthe first high-resolution image, the pixel (attention pixel) being thetarget of the calculation of the permissible amplitude. As illustratedin FIG. 4, a total of 36 pixels 511, 512, 513, 514, . . . , 515, 516,517, 518, . . . , 519 are arranged in a matrix of horizontal 6 pixelsand vertical 6 pixels selected from among a plurality of pixelsconstituting the low-resolution image such that the pixel 501 isarranged approximately at the center of these pixels, and the total of36 pixels 511, 512, 513, 514, . . . , 515, 516, 517, 518, . . . , 519constitute an area S 500.

The feature generating unit 103 selects the total of 36 pixels 511, 512,513, 514, . . . , 519 from the low-resolution image to be included inthe area S 500 such that the attention pixel, namely the pixel 501 isarranged approximately at the center of the 36 pixels. Next, the featuregenerating unit 103 divides the area S 500 into four areas 502, 503, 504and 505. Each of the areas 502, 503, 504 and 505 includes a total ofnine pixels of the low-resolution image arranged in a matrix ofhorizontal 3 pixels and vertical 3 pixels. The area 502 is adjacent tothe upper left side of the pixel 501, the area 503 is adjacent to theupper right side of the pixel 501, the area 504 is adjacent to the lowerleft side of the pixel 501, and the area 505 is adjacent to the lowerright side of the pixel 501. Note that a pixel 515 is arranged at aposition nearest to the pixel 501 among the nine pixels of thelow-resolution image included in the area 502, a pixel 516 is arrangedat a position nearest to the pixel 501 among the nine pixels of thelow-resolution image included in the area 503, a pixel 517 is arrangedat a position nearest to the pixel 501 among the nine pixels of thelow-resolution image included in the area 504, and a pixel 518 isarranged at a position nearest to the pixel 501 among the nine pixels ofthe low-resolution image included in the area 505.

The feature generating unit 103 selects a pixel having the maximum pixelvalue among the nine pixels of the low-resolution image included in thearea 502, and recognizes the pixel value of the selected pixel as MAX1.Also, the feature generating unit 103 selects a pixel having the minimumpixel value among the nine pixels of the low-resolution image includedin the area 502, and recognizes the pixel value of the selected pixel asMIN1. In this way, the feature generating unit 103 calculates a pair ofthe maximum value MAX1 and the minimum value MIN1 from the area 502.Similarly, the feature generating unit 103 calculates a pair of maximumvalue MAX2 and minimum value MIN2 from the area 503, a pair of maximumvalue MAX3 and minimum value MIN3 from the area 504, and a pair ofmaximum value MAX4 and minimum value MIN4 from the area 505.

It is assumed here that the pair of the maximum value MAX1 and theminimum value MIN1 is representative of the area 502, and that themaximum value MAX1 and the minimum value MIN1 are arranged at the sameposition as the pixel 515 in the area 502. Similarly, it is assumed thatthe maximum value MAX2 and the minimum value MIN2 are arranged at thesame position as the pixel 516 in the area 503, the maximum value MAX3and the minimum value MIN3 are arranged at the same position as thepixel 517 in the area 504, and the maximum value MAX4 and the minimumvalue MIN4 are arranged at the same position as the pixel 518 in thearea 505.

Based on the above assumption, FIG. 5 illustrates the positionalrelationship among the maximum values MAX1, MAX2, MAX3, MAX4, and thepixel 501, and FIG. 6 illustrates the positional relationship among theminimum values MIN1, MIN2, MIN3, MIN4, and the pixel 501.

As illustrated in FIG. 5 as one example, the feature generating unit 103calculates the maximum value MAX of the permissible amplitude of theattention pixel 501 by linearly interpolating MAX1, MAX2, MAX3 and MAX4by using the positional relationship among the maximum values MAX1,MAX2, MAX3, MAX4, and the attention pixel 501, and the decimal parts pxand py of the interpolation phase of the attention pixel 501.

Also, as illustrated in FIG. 6 as one example, the feature generatingunit 103 calculates the minimum value MIN of the permissible amplitudeof the attention pixel 501 by linearly interpolating MIN1, MIN2, MIN3and MIN4 by using the positional relationship among the minimum valuesMIN1, MIN2, MIN3, MIN4, and the attention pixel 501, and the decimalparts px and py of the interpolation phase of the attention pixel 501.

As described above, the feature generating unit 103 calculates themaximum value MAX and the minimum value MIN of the permissible amplitudefor each of all the pixels constituting the first high-resolution image.

(ii-2) Second Example

The second example for calculating the permissible amplitude isexplained with reference to FIGS. 7-8.

In the second example for calculating the permissible amplitude, thefeature generating unit 103 corrects the maximum value MAX and theminimum value MIN calculated by the aforementioned first example, basedon the concave-up/down change of the pixel values in the area S.

FIGS. 7 and 8 illustrate examples of changes of pixel values in the areaS in the x-axis direction. FIG. 7 illustrates a case where, the changealong the x axis of the pixel values of a plurality of pixels arrangedin the x-axis direction forms a concave-down shape in the area S. FIG. 8illustrates a case where, the change along the x axis of the pixelvalues of the pixels arranged in the x-axis direction forms a concave-upshape in the area S. Note that although FIGS. 7 and 8 illustrate onlychanges of the pixel values in the x-axis direction for the sake ofsimplification, it may also be assumed that the pixel values of aplurality of pixels arranged in the y-axis direction change in a similarmanner. It may also be assumed that, in the whole space of the area S,the change of pixel values of a plurality of pixels arranged in the areaS forms a concave-down or concave-up shape.

The feature generating unit 103 judges whether, in the whole space ofthe area S, the change of pixel values of a plurality of pixels arrangedin the area S forms a concave-down shape or a concave-up shape.

When it judges that the change of pixel values in the whole space of thearea S forms a concave-down shape, the feature generating unit 103calculates a corrected minimum value MIN′ by multiplying a coefficientα, which is a value not greater than 1, by the minimum value MIN of thepermissible amplitude calculated by the first example for calculatingthe permissible amplitude, based on the following Equation 12.

MIN′=α×MIN  (Equation 12)

Also, when it judges that the change of pixel values in the whole spaceof the area S forms a concave-up shape, the feature generating unit 103calculates a corrected maximum value MAX′ by multiplying a coefficientα, which is a value not smaller than 1, by the maximum value MAX of thepermissible amplitude calculated by the first example for calculatingthe permissible amplitude, based on the following Equation 13.

MAX′=α×MAX  (Equation 13)

In this way, the permissible amplitude is adjusted in compliance withthe features of the low-resolution image, by correcting MIN and MAXbased on whether the change of pixel values in the whole space of thearea S forms a concave-down shape or a concave-up shape. This makes itpossible to, in the low-resolution image, increase the maximum valuewith regard to a portion where the change of pixel values forms aconcave-up shape, and decrease the minimum value with regard to aportion where the change of pixel values forms a concave-down shape,thereby realizing restriction of the amplitude of the peripheral valuesaround the attention pixel of the low-resolution image, exceeding themaximum and minimum values.

Note that the coefficient α is a function of the concave-down orconcave-up shape in the area S, represented by the following Equation14.

α=f(u)  (Equation 14)

In the function f, α may be obtained from a result of applying Laplacianfilter to the area S, for example. Also, a may be obtained from theratio of each pixel value, which is the result of applying a low-passfilter to each pixel value in the area S, to said each pixel value inthe area S.

More specifically, the coefficient α may be determined by multiplying anappropriate coefficient β by a result of applying Laplacian filter. Inthis case, a may become too great depending on the coefficient β or thepixel values. (Each pixel value may be a value in the range from 0 to255. When the pixel values are normalized, each pixel value isrepresented by a value in the range from 0 to 1.0, where 0 indicatesblack and 1.0 indicates white.) When α becomes too great or too small, αis clipped by multiplying α by a clip value. The clip value may be inthe range from 0.5 times to 1.5 times or in the range from 0.25 times to1.75 times, for example. It should be noted here that the clip value isnot limited to this range.

The coefficient α may be a ratio B/A, where A is a pixel value obtainedas a result of applying a low-path filter, and B is each pixel value insmall areas in the area S. Note that in this case, a clipping of thesame level as the Laplacian filtering may be performed as well.

As described above, in the second example for calculating thepermissible amplitude, corrected maximum value and minimum value arecalculated as the permissible amplitude, based on the concave-up/downchange of the pixel values in the area S.

(4) Difference Calculating Unit 106

The difference calculating unit 106 receives the first high-resolutionimage from the enlargement interpolation unit 105, and the secondhigh-resolution image from the super resolution enlargement unit 104.

Next, the difference calculating unit 106 calculates a difference valuebetween pixel values of corresponding pixels of the first and secondhigh-resolution images, for each of the pixels constituting the firstand second high-resolution images, based on Equation 13, and outputs thedifference values obtained by the calculation to the adjustment unit107.

Difference value (x,y)=(pixel value of pixel (x,y) in secondhigh-resolution image)−(pixel value of pixel (x,y) in firsthigh-resolution image)  (Equation 15)

Furthermore, the difference calculating unit 106 outputs the receivedsecond high-resolution image to the adjustment unit 107.

With the difference values calculated as described above, it is possibleto extract a portion representing a change between the secondhigh-resolution image output from the super resolution enlargement unit104 and the first high-resolution image output from the enlargementinterpolation unit 105. Noises that occur in the super resolutionprocess are included in the difference values. Thus in the followingprocess, the difference values are corrected to restrict the noises.

(5) Adjustment Unit 107

The adjustment unit 107 receives the second high-resolution image andall the calculated difference values (x,y) from the differencecalculating unit 106, and receives the features from the featuregenerating unit 103. As described above, the feature of each of thepixels constituting the first high-resolution image is the gradientinformation, or the permissible amplitude, or a combination of thegradient information and the permissible amplitude.

Next, the adjustment unit 107 obtains a corrected difference value (x,y)by correcting a difference value (x,y) calculated by the differencecalculating unit 106 for each of all the received difference values(x,y), based on the following Equation 16.

Corrected difference value (x,y)=difference value (x,y) x gainβ  (Equation 16)

The adjustment unit 107 then outputs all of the calculated correcteddifference values (x,y) to the combination unit 108.

The following describes the method for correcting the difference value(x,y) in detail.

(a) First Correction Method

The adjustment unit 107, as a first correction method, may correct thedifference value by using the gradient information received as thefeature from the feature generating unit 103. In the correction usingthe gradient information, the difference value is corrected based on thesize of the gradient.

The adjustment unit 107 stores, in advance, a correspondence table (notillustrated) that indicates the relationship between the gradient andthe gain β. The correspondence table includes a plurality of pieces ofcorrespondence data which each include a gradient and a gain β. Withthis structure, each piece of correspondence data indicates thecorrespondence between the gradient and the gain β included therein, andwhen a gradient is specified, a gain β corresponding to the specifiedgradient can be used. The gain β is set approximately within the rangefrom 0 times to 16 times.

The adjustment unit 107 obtains a gain β corresponding to a gradientindicated by the gradient information (the feature) by reading it fromthe correspondence table. When a gradient indicated by the gradientinformation is not present in the correspondence table, a gain β thatcorresponds to the gradient indicated by the gradient information may becalculated by linear interpolation by using the plurality of pieces ofcorrespondence data included in the correspondence table.

Note that the adjustment unit 107 may not include the correspondencetable, but may have a formula that represent the relationship betweenthe gradient and the gain β, and calculate the gain β that correspondsto the gradient, in accordance with the formula.

In this way, the adjustment unit 107 obtains a gain β and calculates acorrected difference value by multiplying the obtained gain β by adifference value, as defined in Equation 16.

(First Specific Example of Relationship Between Gradient and Gain β)

FIG. 9A illustrates a first specific example of the relationship betweenthe gradient and the gain β in the correspondence table. A graph 600illustrated in FIG. 9A represents the relationship between the gradientand the gain β, the horizontal axis representing the gradient, thevertical axis representing the gain. In the graph 600, in a range 601where the gradient is smaller than a gradient value 611, the setting ismade such that the gain β is smaller than a gain value 612 and is almostconstant. On the other hand, in a range 602 where the gradient isgreater than the gradient value 611, the setting is made such that thegain β is greater than the gain value 612 and becomes greater as thegradient becomes greater.

As described above, in the case illustrated in FIG. 9A, the gain β isset to be small in a range where the gradient is small, and in a rangewhere the gradient is great, the gain β is set to increase as thegradient increases.

It is considered that, in a range with small gradient, for example, inthe range 601 of the graph 600, the image has a fine texture, and in arange with great gradient, for example, in the range 602 of the graph600, the image has an edge component.

By setting the relationship between the gradient and the gain β asillustrated in FIG. 9A, it is possible to correct the difference valuefor a fine texture by using a small gain β, namely correct thedifference value by a very small amount, and on the other hand, tocorrect the difference value for an edge component such that thestronger the edge component is, the greater the used gain β is, namely,to correct only strong edge components. This makes it possible toenhance only edge components included in an image, without enhancingnoise components included in small amplitudes in the image.

(Second Specific Example of Relationship Between Gradient and Gain β)

Next, FIG. 9B illustrates a second specific example of the relationshipbetween the gradient and the gain β in the correspondence table. A graph620 illustrated in FIG. 9B represents the relationship between thegradient and the gain β, the horizontal axis representing the gradient,the vertical axis representing the gain. The graph 620 indicates that,in a range 621 where the gradient is smaller than a gradient value 631,the setting is made such that the gain β is smaller than a gain value641 and is almost constant. Also, in a range 622 where the gradient isgreater than the gradient value 631 and smaller than a gradient value632, the setting is made such that the gain β increases once and thendecreases. That is to say, in the range 622, as the gradient increases,the gain β increases drastically from the gain value 641 to a gain value643, the peak value, and then decreases drastically from the gain value643 to a gain value 642. Furthermore, in a range 623 where the gradientis greater than the gradient value 632, the setting is made such thatthe gain β decreases smoothly as the gradient increases.

As described above, for the case illustrated in FIG. 9B, the setting ismade such that, in a range where the gradient is small, the gain β iskept to be small; in a range where the gradient is present to a certainextent, the gain β is increased; and in a range where the gradient isgreat, the gain β is decreased to be small again.

By setting the relationship between the gradient and the gain β asillustrated in FIG. 9B, in the range 622 where the image is consideredto include a fine texture, the gain is increased so that the effect ofthe fine-texture portion is maintained and the effect of the ranges 621and 623 other than the range 622 is made small.

As described above, in correcting the difference value by using thegradient information, corrected difference values are obtained bydetermining values of gain β depending on the gradient of the area S,namely, the flatness of the area S, and multiplying the difference valueby the determined values of gain β. The difference values are correctedin this way.

Note that the gain β is basically a value not greater than 1, but may beset to a value not smaller than 1 to enhance an area having a certaingradient. The gain β may be set approximately within the range from 0times to 16 times.

(b) Second Correction Method

The adjustment unit 107 may adopt a second correction method in whichthe difference value is corrected by using the permissible amplitudereceived as the feature from the feature generating unit 103. In thiscorrection using the permissible amplitude, a correction is made withregard to a portion exceeding the permissible amplitude.

The adjustment unit 107 repeatedly performs the following processes (i)to (v) for each of all the pixels constituting the received secondhigh-resolution image.

(i) Select one pixel from the second high-resolution image and extractthe pixel value of the selected pixel.

(ii) Select a permissible amplitude (the maximum value and the minimumvalue) corresponding to the selected pixel, from received permissibleamplitudes.

(iii) Judge whether or not the extracted pixel value is within a rangethat is defined by the selected maximum value and minimum value. That isto say, judge whether or not the extracted pixel value satisfies thecondition that the extracted pixel value is not greater than theselected maximum value and is not smaller than the selected minimumvalue.

(iv) When it is judged that the extracted pixel value is not within therange, calculate a corrected difference value by multiplying the gain βby a difference value corresponding to the selected pixel, and outputthe calculated corrected difference value.

(v) When it is judged that the extracted pixel value is within therange, output the original difference without multiplying the gain β bya difference value corresponding to the selected pixel.

The following describes the second correction method with reference toFIGS. 10 and 11.

In FIG. 10, the horizontal axis represents the x coordinate and thevertical axis represents the pixel value. Furthermore, in FIG. 10, 1101represents a change in pixel value over pixels arranged at positionsindicated by a specific y coordinate and x coordinates 701, 702, . . . ,707, in the second high-resolution image output from the superresolution enlargement unit 104. Furthermore, 1102 and 1103 respectivelyrepresent changes in the maximum value and the minimum value of thepermissible amplitude over the positions indicated by the specific ycoordinate and x coordinates 701, 702, . . . , 707.

In FIG. 10, pixel values of pixels arranged at positions indicated bythe specific y coordinate and x coordinates 701, 702, 703, and 707 arewithin the range of the maximum and minimum values at the correspondingpositions. On the other hand, pixel values of pixels arranged atpositions indicated by the specific y coordinate and x coordinates 704and 705 are greater than the maximum values at the correspondingpositions.

Accordingly, the adjustment unit 107 calculates corrected differencevalues by multiplying the gain β by difference values calculated for thepositions indicated by the specific y coordinate and x coordinates 704and 705, and outputs the calculated corrected difference values.

Also, the adjustment unit 107 outputs the original difference values asthe corrected difference values with regard to the positions indicatedby the specific y coordinate and x coordinates 701, 702, 703, 706 and707.

The combination unit 108, which is described later, generates the thirdhigh-resolution image by using the corrected difference valuescalculated as described above, and outputs the third high-resolutionimage.

In FIG. 11, the horizontal axis represents the x coordinate and thevertical axis represents the pixel value. FIG. 11 illustrates a change1201 in pixel value over pixels arranged at positions indicated by thespecific y coordinate and x coordinates 701, 702, . . . , 707, in thethird high-resolution image generated by the combination unit 108, aswell as the changes 1101, 1102 and 1103 illustrated in FIG. 10.

As illustrated in FIG. 11, compared with the second high-resolutionimage, in the third high-resolution image, the pixel values at thepositions indicated by the specific y coordinate and x coordinates 704and 705 have been corrected downward.

That is to say, the pixel values at the positions have been decreased bythe correction.

In this way, the adjustment unit 107 corrects the difference value withregard to a portion exceeding the permissible amplitude.

(c) Other Correction Methods

The following describes another method for correction.

First, gains β1 and β2 are calculated independently by theabove-described first and second correction methods, and the calculatedgains β1 and β2 are added.

Gain β=gain β1+gain β2

Next, a corrected difference value is calculated by multiplying thedifference by the calculated gain β.

The following describes a further method for correction.

A corrected difference value is calculated by the above-described firstcorrection method, and the calculated corrected difference value isoutput to the combination unit 108. The combination unit 108 generatesthe third high-resolution image by using the corrected difference valueand the first high-resolution image. Subsequently, the adjustment unit107 judges by the second correction method whether or not the thirdhigh-resolution image has a portion exceeding the permissible amplitude,and with regard to pixels exceeding the permissible amplitude,calculates a doubly corrected difference value by multiplying thecorrected difference value by a gain obtained by the second correctionmethod. The combination unit 108 then generates a fourth high-resolutionimage by adding the doubly corrected difference value to the pixelsexceeding the permissible amplitude of the third high-resolution image,and outputs the generated fourth high-resolution image.

(6) Combination Unit 108

The combination unit 108 receives the first high-resolution image fromthe enlargement interpolation unit 105, receives corrected differencevalues from the adjustment unit 107, generates a third high-resolutionimage by adding the corrected difference values to pixel values of thecorresponding pixels of the first high-resolution image, and outputs thegenerated third high-resolution image.

Third high-resolution image (x,y)=first high-resolution image(x,y)+corrected difference value (x,y)

Since the corrected difference value is obtained by correcting thedifference value between corresponding pixels of the first and secondhigh-resolution images for each pixel constituting the high-resolutionimage, the corrected difference value is added to the pixel value ofeach corresponding pixel constituting the first high-resolution image,thereby obtaining the result of the corrected super resolution process.

3. Operation

The following describes the operation of the image processing device 10.

(1) Operation of Image Processing Device 10

The following describes the operation of the image processing device 10with reference to the flowchart illustrated in FIG. 12, centering on theoperation of the image processing unit 100.

The input unit 201 obtains a low-resolution image (step S101); theenlargement interpolation unit 105 generates a first high-resolutionimage by performing the enlargement interpolation process (step S102);the super resolution enlargement unit 104 generates a secondhigh-resolution image by performing the super resolution process (step S103); the feature generating unit 103 generates the feature (step S104);the difference calculating unit 106 calculates the difference value(step S105); the adjustment unit 107 calculates the corrected differencevalue (step S106); the combination unit 108 generates the thirdhigh-resolution image by adding the corrected difference value to thefirst high-resolution image (step S107); and outputs the generated thirdhigh-resolution image (step S108).

(2) Operation of Enlargement Interpolation Unit 105

The following describes the operation of the enlargement interpolationunit 105 with reference to the flowchart illustrated in FIG. 13.

The enlargement interpolation unit 105 calculates the interpolationinterval dx in the horizontal direction and the interpolation intervaldy in the vertical direction, based on the Equations 3 and 4, using thevalues in XS, in YS, outXS, and outYS, the value in XS indicating thenumber of pixels in the horizontal direction and the value in YSindicating the number of pixels in the vertical direction of thelow-resolution image, and the value outXS indicating the number ofpixels in the horizontal direction and the value outYS indicating thenumber of pixels in the vertical direction of the third high-resolutionimage that is to be output (step S302).

The enlargement interpolation unit 105 initializes variable y to 0(S303); initializes variable x to 0 (S304); and obtains values ofperipheral pixels centered on the coordinates (x,y) from thelow-resolution image (S305). The enlargement interpolation unit 105 thencalculates the decimal parts px and py of the x and y coordinates(S306).

Subsequently, the enlargement interpolation unit 105 performs theenlargement interpolation process by using the values px and py and thepixel values of peripheral pixels centered on the point (x,y) of thelow-resolution image (S307); stores, as the output pixel values, thepixel values output from the enlargement interpolation process (S308);and adds dx to the variable x (S309). Here, when the variable x does notexceed the value outXS indicating the number of pixels in the horizontaldirection (No in S310), the control returns to S305 to repeat theprocesses. Also, when the variable x exceeds the value outXS (Yes inS310), the enlargement interpolation unit 105 adds dy to the variable y(S311). When the variable y does not exceed the value outYS (No inS312), the control returns to S304 to repeat the processes. When thevariable y exceeds the value outYS (Yes in S312), the process ends sincepixel values of all pixels constituting the first high-resolution imagehave been output.

4. Conclusion

With execution of the above-described process, the result of the superresolution process is corrected. By making a correction using thegradient information, it is possible to restrict a noise from occurringin a portion where the change of pixel values of pixels in an image isflat, or from occurring in a portion such as an edge where the gradientis great. Also, by making a correction using the permissible amplitude,it is possible to correct a portion where a protruding noise hasoccurred to pixel values of pixels in the low-resolution image that isan input image. By performing the above-described correction processes,it is possible to restrict noises that have occurred in the superresolution process and obtain a visually excellent image.

(Other Modifications)

Up to now, the present invention has been described through anembodiment thereof. However, the present invention is not limited to theembodiment, but includes, for example, the following modifications.

(1) The difference calculating unit 106 may calculate the differencevalue (x,y) based on the following Equation 17 instead of Equation 15.

Difference value (x,y)=(pixel value of pixel (x,y) in firsthigh-resolution image)−(pixel value of pixel (x,y) in secondhigh-resolution image)  (Equation 17)

In this case, the combination unit 108 generates the thirdhigh-resolution image based on the following Equation 18.

Third high-resolution image (x,y)=first high-resolution image(x,y)−corrected difference value (x,y)  (Equation 18)

(2) The super resolution enlargement unit 104 may use the superresolution enlargement process method disclosed in Non-Patent Literature1 or 2.

(3) The following structures are applicable.

According to one aspect of the present invention, there is provided animage processing device for generating a high-resolution image from alow-resolution image, comprising: an obtaining circuit that obtains thelow-resolution image; an enlargement interpolation circuit thatgenerates a first high-resolution image that is higher in resolutionthan the low-resolution image, by performing an enlargementinterpolation on the obtained low-resolution image; a super resolutionprocessing circuit that generates, from the low-resolution image, asecond high-resolution image that is equal in resolution to the firsthigh-resolution image, by tracing back a process through which an imageis assumed to be degraded in resolution, the process being representedby a degradation model; a feature generating circuit that generates afeature for each of a plurality of pixels constituting the firsthigh-resolution image by using the low-resolution image; a differencecalculating circuit that calculates, for each of the plurality of pixelsconstituting the first high-resolution image, a difference value betweena value of that pixel and a value of a corresponding pixel in the secondhigh-resolution image; an adjustment circuit that calculates a correcteddifference value by correcting the calculated difference value by usingthe generated feature; and a combination circuit that generates a thirdhigh-resolution image by adding corrected difference values calculatedby the adjustment circuit to values of corresponding pixels in the firsthigh-resolution image.

According to another aspect of the present invention, there is providedan image processing device for generating a high-resolution image from alow-resolution image, comprising: a memory storing a computer programthat is composed of a plurality of computer instructions; and aprocessor configured to read the computer instructions one by one fromthe computer program stored in the memory, decode the read computerinstructions, and operate in accordance with the decoded computerinstructions. The computer program causes a computer to execute: anobtaining step of obtaining the low-resolution image; an enlargementinterpolation step of generating a first high-resolution image that ishigher in resolution than the low-resolution image, by performing anenlargement interpolation on the obtained low-resolution image; a superresolution processing step of generating, from the low-resolution image,a second high-resolution image that is equal in resolution to the firsthigh-resolution image, by tracing back a process through which an imageis assumed to be degraded in resolution, the process being representedby a degradation model; a feature generating step of generating afeature for each of a plurality of pixels constituting the firsthigh-resolution image by using the low-resolution image; a differencecalculating step of calculating, for each of the plurality of pixelsconstituting the first high-resolution image, a difference value betweena value of that pixel and a value of a corresponding pixel in the secondhigh-resolution image; an adjustment step of calculating a correcteddifference value by correcting the calculated difference value by usingthe generated feature; and a combination step of generating a thirdhigh-resolution image by adding corrected difference values calculatedby the adjustment unit to values of corresponding pixels in the firsthigh-resolution image.

According to a still another aspect of the present invention, there isprovided a computer-readable non-transitory recording medium storing acomputer program for use in an image processing device for generating ahigh-resolution image from a low-resolution image. The computer programcauses the image processing device, which is a computer, to execute: anobtaining step of obtaining the low-resolution image; an enlargementinterpolation step of generating a first high-resolution image that ishigher in resolution than the low-resolution image, by performing anenlargement interpolation on the obtained low-resolution image; a superresolution processing step of generating, from the low-resolution image,a second high-resolution image that is equal in resolution to the firsthigh-resolution image, by tracing back a process through which an imageis assumed to be degraded in resolution, the process being representedby a degradation model; a feature generating step of generating afeature for each of a plurality of pixels constituting the firsthigh-resolution image by using the low-resolution image; a differencecalculating step of calculating, for each of the plurality of pixelsconstituting the first high-resolution image, a difference value betweena value of that pixel and a value of a corresponding pixel in the secondhigh-resolution image; an adjustment step of calculating a correcteddifference value by correcting the calculated difference value by usingthe generated feature; and a combination step of generating a thirdhigh-resolution image by adding corrected difference values calculatedby the adjustment unit to values of corresponding pixels in the firsthigh-resolution image.

(4) Each of the above-described devices may be a computer system thatincludes a microprocessor, ROM, RAM, and hard disk unit. A computerprogram is stored in the RAM or the hard disk unit. The microprocessoroperates in accordance with the computer program, thereby enabling thatdevice to realize its functions. The computer program mentioned above iscomposed of a plurality of instruction codes which each instructs thecomputer to realize a predetermined function.

(5) The present invention may be an image processing method for use inan image processing device. The present invention may be a computerprogram that allows a computer to realize the method, or may be adigital signal representing the computer program.

Furthermore, the present invention may be a computer-readable recordingmedium such as a flexible disk, a hard disk, CD-ROM, MO, DVD, DVD-ROM,DVD-RAM, BD (Blu-ray Disc), or a semiconductor memory, that contains thecomputer program or the digital signal recorded thereon. Furthermore,the present invention may be the computer program or the digital signalrecorded on any of the above-mentioned recording mediums.

Furthermore, the present invention may be the computer program or thedigital signal transmitted via an electric communication line, awireless or wired communication line, a network of which the Internet isrepresentative, or a data broadcast.

Also, the present invention may be a computer system that includes amicroprocessor and a memory, wherein a computer program is stored in thememory, and the microprocessor operates in accordance with the computerprogram.

Furthermore, the computer program or the digital signal may be recordedon any of the above-described recording mediums and transferred in thatform to another independent computer system to be implemented therein,or the computer program or the digital signal may be transferred toanother independent computer system via the network or the like and maybe implemented in the other independent computer system.

(6) The present invention may be any combination of the above-describedembodiments and modifications.

INDUSTRIAL APPLICABILITY

When a degradation model used in the super resolution process differsfrom a real degradation process of an obtained low-resolution image, theimage processing device of the present invention can obtain a visuallyexcellent high-resolution image by restricting noises from occurring,and thus is suitable for an image processing technology for generating ahigh-resolution image from a low-resolution image.

REFERENCE SIGNS LIST

-   -   101 low-resolution image    -   102 third high-resolution image    -   103 feature generating unit    -   104 super resolution enlargement unit    -   105 enlargement interpolation unit    -   106 difference calculating unit    -   107 adjustment unit    -   108 combination unit    -   201 input unit    -   202 tuner    -   203 external terminal    -   204 memory card    -   205 memory    -   206 processor    -   207 decoding unit    -   208 display control unit    -   209 video driver    -   210 hard disk    -   211 display device

1-7. (canceled)
 8. An image processing device for generating ahigh-resolution image from a low-resolution image, comprising: anobtaining unit configured to obtain the low-resolution image; anenlargement interpolation unit configured to generate a firsthigh-resolution image that is higher in resolution than thelow-resolution image, by performing an enlargement interpolation on thelow-resolution image; a super resolution processing unit configured togenerate, from the low-resolution image, a second high-resolution imagethat is equal in resolution to the first high-resolution image, bytracing back a process through which an image is assumed to be degradedin resolution, the process being represented by a degradation model; afeature generating unit configured to generate a feature for eachposition of a plurality of pixels constituting the first high-resolutionimage by using the low-resolution image; a difference calculating unitconfigured to calculate, for each of the plurality of pixelsconstituting the first high-resolution image, a difference value betweena value of that pixel and a value of a corresponding pixel in the secondhigh-resolution image; an adjustment unit configured to calculate acorrected difference value by correcting the calculated difference valueby using the generated feature; and a combination unit configured togenerate a third high-resolution image by adding corrected differencevalues calculated by the adjustment unit to values of correspondingpixels in the first high-resolution image, wherein the featuregenerating unit generates, as the feature, gradient informationindicating a gradient of a value of a pixel in an image area in thelow-resolution image corresponding to peripheral pixels of each of theplurality of pixels constituting the first high-resolution image, thegradient information being indicated by a norm of the image area, andthe adjustment unit corrects the calculated difference value by usingthe generated gradient information.
 9. The image processing device ofclaim 8, wherein the adjustment unit calculates a gain that increases ordecreases depending on the gradient information, and calculates thecorrected difference value by multiplying the calculated gain by thedifference value.
 10. The image processing device of claim 8, whereinthe feature generating unit generates, as the feature, a permissiblerange from an image area in the low-resolution image corresponding toperipheral pixels of each of the plurality of pixels constituting thefirst high-resolution image, the permissible range being a range ofvalues that can be taken by that pixel, and the adjustment unit correctsthe calculated difference value for each pixel in the secondhigh-resolution image that has a value exceeding the permissible range.11. An image processing method for use in an image processing device forgenerating a high-resolution image from a low-resolution image,comprising: an obtaining step of obtaining the low-resolution image; anenlargement interpolation step of generating a first high-resolutionimage that is higher in resolution than the low-resolution image, byperforming an enlargement interpolation on the low-resolution image; asuper resolution processing step of generating, from the low-resolutionimage, a second high-resolution image that is equal in resolution to thefirst high-resolution image, by tracing back a process through which animage is assumed to be degraded in resolution, the process beingrepresented by a degradation model; a feature generating step ofgenerating a feature for each position of a plurality of pixelsconstituting the first high-resolution image by using the low-resolutionimage; a difference calculating step of calculating, for each of theplurality of pixels constituting the first high-resolution image, adifference value between a value of that pixel and a value of acorresponding pixel in the second high-resolution image; an adjustmentstep of calculating a corrected difference value by correcting thecalculated difference value by using the generated feature; and acombination step of generating a third high-resolution image by addingcorrected difference values calculated by the adjustment unit to valuesof corresponding pixels in the first high-resolution image, wherein thefeature generating step generates, as the feature, gradient informationindicating a gradient of a value of a pixel in an image area in thelow-resolution image corresponding to peripheral pixels of each of theplurality of pixels constituting the first high-resolution image, thegradient information being indicated by a norm of the image area, andthe adjustment step corrects the calculated difference value by usingthe generated gradient information.
 12. A computer-readablenon-transitory recording medium storing a computer program for imageprocessing for use in an image processing device for generating ahigh-resolution image from a low-resolution image, the computer programcausing a computer to execute: an obtaining step of obtaining thelow-resolution image; an enlargement interpolation step of generating afirst high-resolution image that is higher in resolution than thelow-resolution image, by performing an enlargement interpolation on thelow-resolution image; a super resolution processing step of generating,from the low-resolution image, a second high-resolution image that isequal in resolution to the first high-resolution image, by tracing backa process through which an image is assumed to be degraded inresolution, the process being represented by a degradation model; afeature generating step of generating a feature for each position of aplurality of pixels constituting the first high-resolution image byusing the low-resolution image; a difference calculating step ofcalculating, for each of the plurality of pixels constituting the firsthigh-resolution image, a difference value between a value of that pixeland a value of a corresponding pixel in the second high-resolutionimage; an adjustment step of calculating a corrected difference value bycorrecting the calculated difference value by using the generatedfeature; and a combination step of generating a third high-resolutionimage by adding corrected difference values calculated by the adjustmentunit to values of corresponding pixels in the first high-resolutionimage, wherein the feature generating step generates, as the feature,gradient information indicating a gradient of a value of a pixel in animage area in the low-resolution image corresponding to peripheralpixels of each of the plurality of pixels constituting the firsthigh-resolution image, the gradient information being indicated by anorm of the image area, and the adjustment step corrects the calculateddifference value by using the generated gradient information.